Articulated objects such as the human body, human hand, a laptop computer, a robot, an animal, or other articulated object, are challenging to track with high levels of accuracy and speed from captured sensor data such as video images, depth camera images and other sensor data. Tracking such articulated objects is complex for many reasons, including that the object being tracked has its own particular shape and this shape varies between different objects in the same class of objects. For example, a human hand is a class of objects and within that class, there are many different individual human hands with different shapes. A laptop computer is also a class of objects and within the class are lots of different designs of laptop computers having different shapes.